Handwriting information processing apparatus, handwriting information processing method, and storage medium having program stored therein for handwriting information processing

ABSTRACT

A handwriting information processing apparatus is provided, having: a coordinate input part that detects a handwriting input made by a user and outputs a coordinate value; a processing part that performs predetermined processes based on the above coordinate value output from the coordinate input part; and a display part that displays an output of the processing part. The processing part further includes a coordinate value storing part that stores all of the coordinate values output by the coordinate input part along with stroke information and line information of the handwriting input; and a handwritten character recognition part that recognizes handwritten characters in a batch based on the coordinate values, the stroke information, and the line information stored in the coordinate value storing part.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates generally to a handwritinginformation processing technique, and more particularly, the presentinvention relates to a handwriting information processing technique forallowing the user to input handwritten characters and convert them intotyped text at a desired timing so as to be displayed on a displayapparatus.

[0003] 2. Description of the Related Art

[0004] Examples of a handwriting information processing apparatus havingmeans for recognizing handwriting in the related art are describedbelow.

[0005] (1) In Japanese Patent Laid-Open No. 2000-099223, a dataprocessor (handwriting information processing apparatus) having aninterface with fixed handwriting input spaces wherein recognition errorscan be easily corrected is disclosed. In this data processor, acharacter code corresponding to the handwriting is sent to the relevantapplication.

[0006] (2) In Japanese Patent Laid-Open No. 2001-014103, a characterinput device (handwriting information processing apparatus) having aninterface with fixed handwriting input spaces, as in the invention of(1), wherein an input method can be switched in an efficient manner isdisclosed.

[0007] (3) In Japanese Patent Laid-Open No. 2000-076380, a handwritinginput device (handwriting information processing apparatus) that allowsinputting of the handwritten characters in any direction without havingto consider fixed input spaces or designated text format upon inputtinghandwritten characters on a touch panel is disclosed.

[0008] (4) In Japanese Patent Laid-Open No. 09-319503, a text processingapparatus (handwriting information processing apparatus) that can beeasily manipulated to perform text inputting and text editing using acombination of a written character string input method wherein thehandwritten characters do not go through a recognition process and arehandled as stroke (a plurality of points connecting one end to the otherin the, handwritten character) data, and a recognized character stringinput method wherein the handwritten characters are recognized andconverted into character code is disclosed.

[0009] (5) In Japanese Patent Laid-Open 2000-043485, an electronicblackboard (handwriting information processing apparatus) for use in aconference and the like, wherein handwritten characters input to a touchpanel are processed for character recognition, converted into typedtext, and displayed on the touch panel is disclosed. According to thisinvention, neat typed text can be obtained in place of a sloppyhandwritten version of the topic and minutes of a conference, forexample.

[0010] The above descriptions of the related art are samples of priorinventions relating to handwritten character recognition apparatuses,particularly, to the user interface of the apparatus. The handwrittencharacter recognition user interface can be roughly divided into twotypes. One type is for the likes of inventions (1) and (2) wherein fixedinput spaces are provided and the user is required to input onecharacter in each space. This type is the most conventional userinterface. In this type of user interface, it is unnecessary to performa cut-out process for each of the characters and recognition errors dueto an error in the cut-out process can be avoided. However, this type ofinterface imposes a great burden on the user and is far from beinguser-friendly (hereinafter, handwritten character recognition in such atype of user interface is referred to as ‘fixed space handwrittencharacter recognition’). The other type of user interface includes theabove inventions (3) and (4), wherein no fixed input spaces are providedso that the user can write freely into the handwriting input region, anda cut-out process for each of the characters is performed in order torecognize the handwritten characters. In this type of user interface,recognition error does occur as a result of the character cut-outprocess; however, this type of interface is advantageous in that theuser can input characters with the ease of writing with a conventionalpencil (hereinafter, the handwritten character recognition in such atype of user interface is referred to as ‘free handwritten characterrecognition’).

[0011] The conventional methods of handwritten character recognition arethe above-described fixed space handwritten character recognition andthe free handwritten character recognition. In these conventionalrecognition methods, although conversion into typed text is performedbased on information on the shapes of the hand written characters, noprocess is undertaken concerning the position at which the charactersare written nor the size thereof. Thus, in the conventional method, thehandwritten characters input by the user are displayed at apredetermined character position and converted into a code character ofthe same size regardless of the size or the position in which theoriginal handwritten characters have been input. For example, in ahandwritten text of the minutes of a conference and the like, the usermay want to emphasize a word/phrase, to enlarge the characters of animportant word/phrase, or leave a blank to be filled in later In theconventional art, it is impossible to perform all of these operations inthe initial text conversion process. Thereby, the user has to performindividual processes after the text conversion process for changing thefont size of a particular word/phrase or moving the existing charactersby inputting spaces in the desired area to add a word/phrase.

[0012] Since performing such operations imposes a great burden on theuser, it is preferable that the conversion process be simpler. Thus, ahandwriting information processing apparatus with a user interface thatis capable of reflecting the intentions of the user inputting thehandwritten characters, such as displaying typed characters in a largefont for the corresponding handwritten characters that should beemphasized or providing a blank space between characters.

[0013] Also, when the user is able to write freely into the input regionof the handwriting information processing apparatus, it is difficult torealize 100% character recognition, that is, a 100% probability of thehandwritten characters being converted to the code characters originallyintended by the user, due to the handwriting peculiarities of each user.Thus, it is necessary to edit the code characters displayed in responseto the handwritten character input. This also imposes a burden on theuser, and therefore, it is preferable that very simple operations enablevarious editing processes. Also, editing operations that are intuitivein nature and easily associated or unexpected and amusing are morelikely to be accepted by the user since the learning of these variousediting operations can be facilitated.

[0014] In a conference and the like, it is imperative that importantsubjects and propositions that are discussed be organized inwell-devised minutes so that all members of the conference are able tocomprehend the discussion in a restricted time period. Thus, in the userinterface of a handwriting information processing apparatus used in aconference such as an electronic blackboard, operations such asemphasizing a heading, adding a postscript, and the like should be madesimple for the user.

[0015] Further in the conventional art, recognition processing isperformed for each character or for a string of characters accumulateduntil a certain length is reached, after which the recognized characterstring is shown and the user determines the desired character (changesto the character string). This method of progressively converting thehandwritten characters is not very suitable for use in a conference.That is, the necessity of checking the recognition results of thehandwritten characters right in the middle of a discussion may arise,which may result in an interruption in the flow of thoughts. Thus, inconverting handwritten information during a conference into type text,the handwritten information is preferably saved as coordinate values,wherein the actual conversion of the information into typed text iscompleted when the discussion sinners down or at the end of theconference.

[0016] Additionally, in a conference, the positioning of the writtencharacter strings is quite important and a character string usually hasa meaningful relationship with the character strings written above andbelow it. Thus, it is preferable that the outline of the characterstrings that are represented as a list or in an indent format, forexample, be represented as they are written. Further, if the outlinedcharacter strings can be easily saved in a file or sent to anotherperson, the information generated from the conference can be put toeffective use.

[0017] Also, in the conventional method, when a user wants to converthandwritten characters input as ‘hiragana’ characters into Chinesecharacters, the user has to take an action such as pushing a Chinesecharacter conversion button and the like. In such a case, the IME (InputMethod Editor) by Windows (registered trademark) performs Chinesecharacter conversion only for the ‘hiragana’ character portion of thecharacter string even when it contains both ‘hiragana’ and Chinesecharacters, thereby leaving the portion originally handwritten inChinese characters unconverted. Thus, it is preferable that the Chinesecharacter conversion and the handwriting information processing intotyped text be performed simultaneously.

SUMMARY OF THE INVENTION

[0018] The present invention has been developed in response to theabove-mentioned problems of the related art, and its object is toprovide a handwriting information processing apparatus, a handwritinginformation processing method, a handwriting information processingprogram, and a storage medium storing the handwriting informationprocessing program, wherein adapted characters are displayed accordingto the size and position of a handwritten character input, editingoperations of the adapted characters are arranged to be simple andamusing, handwritten character input and editing operations are realizedin an efficient manner, and a user interface with user-friendly featuresenabling easy learning of its operation is provided. Accordingly, in thepresent invention, a character recognition process based on ahandwritten character input to an information input region, theselection of a character code corresponding to the handwrittencharacter, a character adaptation process on a code character assignedto the character code based on an input state of the handwrittencharacter, and display of the obtained adapted character from thecharacter adaptation process on a display apparatus are performed.

[0019] It is a further object of the present invention to provide ahandwriting information processing apparatus wherein handwritteninformation is stored along with the stroke information and the lineinformation and converted in a batch at arbitrary timing chosen by theuser rather than being progressively converted into typed text, and anoutline of the character strings is drafted by obtaining the indents ofthe character strings based on information on the positions of thehandwriting input. Accordingly, a handwriting information processingapparatus of the present invention includes:

[0020] a coordinate input part that detects a handwriting input made bya user and outputs a coordinate value;

[0021] a processing part that performs predetermined processes based onthe coordinate value output from the coordinate input part; and

[0022] a display part that displays an output of the processing part;

[0023] wherein the processing part includes:

[0024] a coordinate value storing part that stores all of the coordinatevalues output by the coordinate input part along with stroke informationand line information of the handwriting input;

[0025] a handwritten character recognition part that recognizeshandwritten characters in a batch based on the coordinate values, thestroke information, and the line information stored in the coordinatevalue storing part;

[0026] a Chinese character conversion part that performs a Chinesecharacter conversion by handing a character string obtained by thehandwritten character recognition part to an Input Method Editor;

[0027] an outline drafting part that calculates an indent position ofeach line based on the coordinate values and the line information storedin the coordinate value storing part, obtains outline information of acharacter string, and drafts an outline;

[0028] a result storing part that stores the character string obtainedby the handwritten character recognition part and the outlineinformation obtained by the outline drafting part; and

[0029] an edit processing part that corrects the outline informationbased on information stored in said result storing part.

[0030] It is also an object of the present invention to enable thestorage of the above drafted outline of character strings in a storagemedium, or to enable its transmission to another person via electronicmail by entering an e-mail address and a title. Accordingly the abovehandwriting information processing apparatus further includes:

[0031] an outline draft storing part that stores the character string ofthe drafted outline in a storage medium; and

[0032] an outline draft transmitting part that converts the characterstring of the drafted outline for e-mail transmission., and sends theconverted outline draft to a designated e-mail address.

[0033] A further object of the present invention is to provide a storagemedium that stores a program readable by a central processing unit thatperforms a handwriting information processing method, wherein thehandwriting information processing method includes steps of storinghandwritten information along with the stroke information and the lineinformation, performing a batch-conversion on the handwritteninformation at arbitrary timing chosen by the user rather thanprogressively converting the handwritten characters into typed text,drafting an outline of the character strings by obtaining the indentsbased on information on the positions of the handwriting input, andcorrecting the above drafted outline of the character strings.

[0034] Further, it is an object of the present invention to provide aprogram that executes a handwriting information processing methodincluding steps of storing handwritten information along with the strokeinformation and the line information, performing a batch conversion onthe handwritten information at arbitrary timing chosen by the userrather than progressively converting the handwritten characters intotyped text, drafting an outline of the character strings by obtainingthe indents based on information on the positions of the handwritinginput, and correcting the above drafted outline of the characterstrings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0035]FIGS. 1A and 1B are diagrams illustrating exemplary outlineconfigurations of a handwriting information processing apparatusaccording to the present invention;

[0036]FIG. 2 is a diagram showing an example of the specification of adisplay apparatus having a handwriting input apparatus that is used inthe handwriting information processing apparatus according to thepresent invention;

[0037]FIG. 3 is a block diagram showing the inside configuration of thecomputer used in the handwriting information processing apparatus of afirst embodiment of the present invention;

[0038]FIG. 4 is a diagram showing a screen display of the handwritinginformation processing apparatus according to the first embodiment whena handwriting information processing program is initiated;

[0039]FIG. 5A and FIG. 5B illustrate input strokes of a handwritinginformation input that are input at differing speeds;

[0040]FIG. 6 is a diagram showing the data composition of arecognition-adaptation result memory;

[0041]FIG. 7 is a diagram for illustrating a character line blockregion;

[0042] FIGS. 8A-8C are diagrams for illustrating a character alignmentprocess;

[0043]FIG. 9 is a diagram for illustrating a character translationprocess, wherein an English translation of a word is indicated on thedisplay screen;

[0044]FIG. 10 is a diagram for illustrating a character translationprocess, wherein the meaning of a word is indicated on the displayapparatus;

[0045]FIG. 11 is a diagram for illustrating a character propertyassociation process, wherein the font size and the character spacing ofthe adapted characters in two different character line blocks on thedisplay apparatus are caused to conform;

[0046]FIGS. 12A and 12B are diagrams for illustrating a gesture editingprocess;

[0047]FIG. 13 is a flow chart for illustrating the operation of thehandwriting information processing apparatus according to the firstembodiment of the present invention;

[0048]FIG. 14 is a flow chart for illustrating the procedures of aninput stroke detection process;

[0049]FIG. 15 is a flow chart for illustrating the procedures of ahandwriting information determination process;

[0050]FIG. 16 is a diagram showing an input stroke that has input pointswithin a character line block and an input stroke that does not have anyinput points in a character line block;

[0051]FIG. 17 is a flow chart for illustrating the procedures of acharacter recognition initiation process;

[0052]FIG. 18 is a flow chart for illustrating the procedures of acharacter recognition process;

[0053]FIG. 19 is a flow chart for illustrating the procedures of acharacter adaptation process;

[0054]FIG. 20 illustrates how a handwritten character line block regionis established;.

[0055]FIG. 21 is a flow chart for illustrating the procedures of anediting process type determination process;

[0056]FIG. 22 shows a state in which a one-character edit pointer and acharacter line block edit pointer are displayed;

[0057]FIG. 23 is a flow chart for illustrating the procedures of aone-character editing process;

[0058]FIG. 24 shows a state in which an editing panel is displayed;

[0059] FIGS. 25A-25D are diagrams illustrating how the one-characterediting process is performed;

[0060]FIG. 26 is a flow chart for illustrating the procedures of acharacter line block editing process;

[0061]FIG. 27 is a diagram for illustrating a font size changing processin the character line block;

[0062]FIG. 28 is a diagram for illustrating a moving process on thecharacter line block;

[0063]FIG. 29 is a flow chart for illustrating the procedures of agesture editing process;

[0064]FIG. 30 is a flow chart for illustrating the handwritinginformation processing apparatus according to a second embodiment of thepresent invention;

[0065]FIG. 31 is a diagram showing a screen display of the handwritinginformation processing apparatus according to the second embodiment whenthe recognition instruction discrimination process is performed.

[0066]FIG. 32 is a flow chart for illustrating a coordinate valuestoring process;

[0067]FIG. 33 is a diagram for illustrating a stroke subdivisionprocess;

[0068]FIG. 34 is a diagram for illustrating a line subdivision process;

[0069]FIGS. 35A, 35B, 35B′, 35C, and 35C′ are diagrams for illustratingthe operation of the line subdivision process;

[0070]FIG. 36 is a diagram showing an example of the coordinate value,the stroke information, and the line information stored in a coordinatevalue memory as a result of a coordinate value storing process;

[0071]FIG. 37 is a flow chart illustrating a handwritten characterrecognition process;

[0072]FIG. 38 is a diagram showing an example of the information storedin a result memory as a result of a handwritten character recognitionprocess;

[0073]FIG. 39 is a flow chart for illustrating an IME Chinese characterconversion process;

[0074]FIG. 40 is a flow chart for illustrating an outline draftingprocess;

[0075]FIG. 41 is a diagram for illustrating a line starting positioncalculation process;

[0076] FIGS. 42A-42C are diagrams for illustrating an indenting process;

[0077]FIG. 43 is a flow chart for illustrating a display process;

[0078]FIG. 44 is a flow chart illustrating the user operationdetermination process;

[0079]FIG. 45 is a diagram showing a screen display of the handwritinginformation processing apparatus according to the second embodiment whenthe user operation determination process is performed;

[0080]FIG. 46 is a flow chart for illustrating, an editing process;

[0081]FIG. 47 is a diagram showing the state of the handwriting inputregion when the editing process is performed;

[0082]FIGS. 48A and 48B are diagrams illustrating an indent correctionprocess; and

[0083]FIG. 49 is a flow chart for illustrating an output process.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0084] In the following, a description of the preferred embodiments ofthe present invention is given with reference to the accompanyingdrawings.

[0085]FIGS. 1A and 1B are diagrams showing exemplary outlineconfigurations of a handwriting information processing apparatusaccording to the present invention that is used in a conference system;and FIG. 2 is a diagram showing an example of the specification of adisplay apparatus having a handwriting input apparatus that is used inthe handwriting information processing apparatus according to thepresent invention.

[0086] As shown in FIGS. 1A and 1B, the handwriting informationprocessing apparatus of the present invention includes a computer 1 asthe handwriting information processing apparatus and a display apparatus2 having a handwriting input apparatus such as the Media Site by RicohLtd, which has the specification shown in FIG. 2. The display apparatus2 having a coordinate input apparatus has a data I/O interface 6 aconnecting it to the computer 1 and the computer 1 has an I/O interface6 b connecting it to the display apparatus 2, as shown in FIG. 1B. Inthe following, the display apparatus 2 having a coordinate inputapparatus is referred to as ‘handwriting input/display apparatus 2’, andthe screen part of this handwriting input/display apparatus 2 isreferred to as ‘handwriting input/display screen 2’. The handwritinginput/display screen 2 is referred to as ‘coordinate input apparatus 2₁’ when used as the coordinate input apparatus, and as ‘displayapparatus 2 ₂’ when used as the display apparatus. That is, thehandwriting input/display screen 2 has display functions for displayingprocessing information and data sent from the computer 1 through the I/Ointerface 6 a and handwriting information inputting functions fordetecting the input positions of handwriting information, for example,by detecting the laser-shielded positions on the screen that has laseremitted thereon from its edges, as is disclosed in Japanese PatentLaid-Open 2000-105671, thereby enabling the user 3 to input handwritinginformation just by lightly touching the screen with an indication part3 ₁ shown in FIG. 1A such as a finger or an input pen. As for thehandwriting input/display apparatus 2, a mouse or a tablet may be usedas the coordinate input apparatus, and a CRT or a projector may be usedas the display apparatus. The handwriting input/display apparatus 2 maybe a combination of a coordinate input apparatus and a projector that isalready in the market such as the Mimio by Virtual Ink and the eBeam byEFI.

[0087] In the following, a description of the configuration of ahandwriting information processing apparatus according to a firstembodiment of the present invention is given.

[0088] The handwriting information processing apparatus according to thefirst embodiment performs a character recognition process and acharacter adaptation process on handwritten characters input to thehandwriting input/display apparatus 2 and displays the results of theseprocesses or edits the results based on a handwriting editing operationinput.

[0089] Herein, the character recognition process is a process ofselecting a code character (character corresponding to a character code)that is determined as having the closest shape to a handwrittencharacter being processed based on a character code conforming to astandard such as the JIS, unicode or ASCII. This process is a publiclyknown process and therefore its description is omitted from thisspecification. The character adaptation process is a process ofadjusting the size of a character to be displayed based on the sizeinformation of the input handwritten character and/or adjusting thedisplay position of the character based on the position information ofthe input handwritten character. A detailed description of this processis given later on.

[0090] With reference to FIG. 3, the computer 1 in the presentembodiment includes the data I/O interface 6 b for inputting data fromthe handwriting input/display apparatus 2 and outputting data to thesame apparatus, a hard disk 8′ having a signal processing unit 7′ and amemory.

[0091] In the above hard disk 8′, a handwriting information processingprogram 9′ is provided. This handwriting information processing program9′ is a program that enables the computer 1 to perform the abovecharacter recognition process and the character adaptation process,which in turn enables the adapted characters, generated from performingthe above character recognition process and adaptation process on ahandwritten character input to the handwriting input screen 2, to bedisplayed on the display apparatus 2 ₂ of the handwriting input screen2, or the editing operation on the adapted characters to be performedbased on the editing operation input at the handwriting input screen 2.

[0092] With the activation of the above program 9, a screen display suchas that shown in FIG. 4 appears on the display apparatus 2 ₂. The abovescreen display of FIG. 4 has an input region (information input region)26 into which handwritten characters and handwritten editing operationscan be input.

[0093] Further, in the upper portion of this screen display, arecognition button 27′, a clear button 28′, a refresh button 29′, ajustification button 30′, a character close-in button 31′, a line widthslider 32′, a mode selection slider 33′, a recognition result displayspace 34′, a change button 35′, a Chinese character conversion button36′, a save button 37′, and an end button 38′ are provided.

[0094] The recognition button 27′ is a button for immediately initiatingthe character recognition process. The clear button 28′ is a button forerasing all the writing displayed in the input region 26′ and forinitializing the contents in a coordinate value memory 10′, arecognition-adaptation result memory 11′, and a character line blockregion memory 13′, which are described in the subsequent paragraphs. Therefresh button 29′ is a button for temporarily deleting the writingdisplayed in the input region 26′ and re-displaying the cleared data.This is used mainly in cases where an inconvenience arises in the screendisplay and the screen display needs to be recovered.

[0095] The justification button 30′ is for justifying thecharacter-spacing of the adapted characters displayed in the inputregion 26′. The character close-in button 31′ is for closing in thecharacter spacing of the adapted characters displayed in the inputregion 26′.

[0096] The line width slider 32′ is for changing the width of a line inwhich the handwriting is input. The mode selection slider 33′ is forselecting the mode in which the character recognition process isperformed, the selection being made depending on which type ofcharacters from among the likes of the ‘hiragana’ characters, the‘katakana’ characters, and the alphabet should be predominantlyrecognized.

[0097] The recognition result display space 34′ is a display space fordisplaying the code character selected as a result of the characterrecognition process. The change button 35′ is for changing the codecharacter with a keyboard, for example, which is not shown in thedrawing, and is used when character recognition based on the handwritinginformation is difficult. The Chinese character conversion button 36′ isfor converting the code characters shown in the recognition resultdisplay space 34′ into Chinese characters.

[0098] The save button 37′ is for saving the adapted characters shown inthe input region 26′ as text data or html data. The end button 38′ isfor ending this handwriting information processing (handwrittencharacter conversion) program.

[0099] The hard disk 8′ includes a coordinate value memory 10′, arecognition-adaptation result memory 11′, a character edit memory 12′, acharacter line block region memory 13′, a character line block regionedit memory 14′, a gesture edit memory 15′, character code data 16′,Chinese character conversion dictionary data 17′, Japanese dictionarydata 18′, and Japanese-English dictionary data 19′.

[0100] The coordinate value memory 10′ is for storing the coordinatevalues of the handwriting information input to the input region 26′.With reference to FIG. 5A, the coordinates of an input starting point22′ at which the user touches the handwriting input screen 2 with aninput pen or the like thereby initiating the handwriting input; and ifthe input point is moved on the input region 26′, the coordinates ofinput points in between the input starting point 22′ and an input endingpoint 24′, namely, the coordinates of partway input points 23′ that aredetected at fixed time intervals (approximately 10 msec intervals) of aninput stroke 25′; and the coordinates of the input ending point 24′ atwhich the input pen is let go from the input region 26′ thereby endingthe handwriting input are stored in the coordinate value memory 10′.

[0101] Since the coordinates of the partway input points 23′ of theinput stroke 25′ are obtained at fixed time intervals, there will befewer partway input points 23′ obtained, as shown in FIG. 5B, when thestroke is input at a fast pace. Thus, based on each of the coordinatesfor the input starting point 22′, the partway input points 23′, and theinput ending point 24′, and the number of the partway input points 23′,information on the input stroke 25′ such as the moving speed, the movingdirection, and the shape can be obtained.

[0102] The recognition-adaptation result memory 11′ is for storing therecognition-adaptation result from the character recognition process andthe character adaptation process performed on the handwritten charactersby the handwriting information processing program 9′. Therecognition-adaptation result memory 11′ contains character data 11p′(‘p’ being a variable) for each character, as shown in FIG. 6. Thecharacter data for each character contains information on the displayproperties of the character such as selected character code data 11 a′indicating the result from selecting the character code closest to thehandwritten character upon performing the character recognition process,candidate character code data 11 b′ indicating the other candidatesconsidered in the selection process, a character line block number 11 c′indicating the line block to which the adapted character correspondingto the handwritten character belongs, a character number 11 d′indicating the position of the adapted character within the abovecharacter line block, position information 11 e′ indicating the positionof the adapted character in the region on the screen, font type 11 f′,font size 11 g′, and font color 11 h′.

[0103] The character line block refers to a set of characters assembledtogether as one block when a series of handwritten characters are input,the block being the unit on which various editing processes areperformed. In this embodiment, when handwritten characters are not inputfor a predetermined period of time, the handwritten characters inputprior to the elapse of this predetermined time period are determined tobelong to a character line block, and the character recognition processis performed in a batch for each of these character line blocks. Thus,in a case where a plurality of handwritten characters are input to thehandwriting input screen 2, the character line block constitutes aplurality of adapted characters corresponding to the above plurality ofhandwritten characters. On the other hand, when handwritten charactersare not input for a predetermined time period after the input of onehandwritten character, the adapted character corresponding to this onehandwritten character becomes a character line block.

[0104] For example, FIG. 6 shows character data 11 p′ for the firsthandwritten character “

” in a series of handwritten ‘hiragana’ characters “

” input as one character line block, the input starting at coordinates(x1, y1) on the input region 26′ and the character line block having aproperty setting in which the font type 11 f′ is Ming-style (Mincho),the font size is 20 points, and the font color is black.

[0105] In a case where “

” is selected as the code character, a JIS character code “0×2422”corresponding to the ‘hiragana’ character “

” is kept in the selected character code data 11 a′. In the candidatecharacter code data 11 b′ a plurality of JIS character codescorresponding to characters that have structures similar to that of thehandwritten character “

” and are candidate characters not selected in the character recognitionprocess such as the JIS character code “0×242a ” for the ‘hiragana’character “

”, and the JIS character code “0×2461” for the ‘hiragana’ character “

” are kept.

[0106] If this character line block corresponding to the series ofcharacters “

” is the third character line block that has been handwritten onto theinput region 26′, the character line block number will be “3”. Also, thecharacter number 11 d′ will be “1” since the character “

” is the first character among the three characters “

” making up this character line block.

[0107] The character edit memory 12′ contains information on thespecific processes of the editing process performed on each of theadapted characters displayed on the display apparatus 2 ₂. In thisembodiment, the character edit memory 12′ contains information on acharacter deletion process, a character insertion process, an adaptedcharacter changing process to another candidate character, and acharacter rewriting process, each of which is associated with apredetermined editing operation input. The above predetermined editingoperation input corresponds to the selection of an edit button in anedit panel, which will be described later.

[0108] Referring to FIG. 7, the character line block region memory 13′contains position information for identifying a character line blockregion 21′, which is a predetermined region surrounding a character lineblock 20′ on the display apparatus 2 ₂. In this embodiment, thecharacter line block region memory 13′ holds information on thecoordinates (xa, ya) of a top left-hand corner point A and thecoordinates (xb, yb) of a bottom right-hand corner point B of thecharacter line block region 21′. When a plurality of character lineblocks are displayed on the display apparatus 2 ₂, information on thecoordinate points of the character line block region for each of thecharacter line blocks is stored in the character line block regionmemory 13′.

[0109] The character line block edit memory 14′ holds information oneach of the specific processes of the editing process performed on theentire character line block 20′. In this embodiment, information on afont size changing process, a character line block moving process, acharacter aligning process, a character translation process in acharacter line block, and a adapted character property associationprocess are stored in the character line block edit memory 14′.

[0110] Each of the editing processes stored in the character line blockedit memory 14′ is associated with an action of inputting an editingoperation to the input region 26′. That is, a specific editing processis performed according to the moving direction, the moving speed, andthe moving distance of an input stroke 25′.

[0111] Herein, the character aligning process refers to the editingprocess of indenting the character line block 20′ displayed at asuitable position on the display apparatus 2 ₂, as shown in FIG. 8A, tothe left side of the input region 26′, as shown in FIG. 8B, or to theright side of the input region 26′, as shown in FIG. 8C. At the sametime, the adapted characters in the character line block 20′ can bepositioned closer to each other (i.e. closing in the space between thecharacters, as shown in FIG. 8B), or justified (i.e. equally spacing outthe characters within the character line block region 21′ whilemaintaining the character line block 20′ at its original length, asshown in FIG. 8C) in the above indent operation.

[0112] Referring to FIG. 9, the translation process of the characters ina character line block is a process of translating a word/phrase formedby the adapted characters displayed on the character line block 20′ intoEnglish based on the Japanese-English dictionary data 19′ and indicatingthe result on the display apparatus 2 ₂. To the contrary, when theword/phrase formed by adapted characters shown in the character lineblock 20′ is in English, a Japanese translation of the above Englishword/phrase can be indicated on the display apparatus 2 ₂ based onEnglish-Japanese dictionary data not shown in the drawing. Also, themeaning of a word/phrase formed by the adapted characters of thecharacter line block 20′ can be displayed on the display apparatus 2 ₂based on the Japanese dictionary data 18′, as shown in FIG. 10.

[0113] The character properties association process is a process ofassociating the character properties of the adapted characters indifferent character line blocks having differing character propertiesfor the adapted characters making up each character line block. Forexample, in a case where two character line blocks 20′ and 39′ havingdifferent font size properties and character spacing properties aredisplayed on the display apparatus 2 ₂, as shown in FIG. 11, the fontsize and the character spacing of the adapted characters “

” making up the character line block 20′ are made to conform with thefont size and the character spacing of the adapted characters “

” making up the character line block 39′ upon moving the character lineblock 20′ into the character line block region of the character lineblock 39′, the two character line blocks then being displayed in anorganized layout. This process is used for such occasions as displayingseparately handwritten character blocks 20′ and 39′ in a well-organizedlist format.

[0114] The gesture edit memory 15′ holds information on the gestureediting processes that correspond to each form of gesture input. Herein,gesture inputs refer to characteristic forms of editing operationinputs, each assigned to a predetermined editing process, provided inorder to facilitate the complicated and numerous editing operationinputs for the character line block 20′.

[0115] For example, in the present embodiment, when a gesture input 54′of slashing from right to left across the adapted characters “

” of the character line block 20′ made up of the adapted characters “

” is made, as shown in FIG. 12A, the adapted characters “

” will be deleted and the character line block will contain the adaptedcharacters “

”. Also, when a gesture input 55′ of encircling the adapted characters “

” of the character line block 20′ made up of the adapted characters “

” is made, as shown in FIG. 12B, the font color of the adaptedcharacters “

” will be changed.

[0116] Other gesture inputs of characteristic shapes such as “

”, “□”, “Δ”, and the like also correspond to various editing processes,and the information on the editing processes corresponding to thesegesture inputs are stored in the gesture edit memory 15′. However, inthis embodiment, the specific descriptions of each of these gestureinputs are omitted.

[0117] The character code data 16′ has character codes conforming to theJapanese Industrial Standards (JIS) and code characters, which are thetyped text characters corresponding to this character code. Uponcharacter recognition, the selected character code data 11 a′ and thecandidate character code data 11 b′ corresponding to a handwrittencharacter are extracted from this character code data 16′.

[0118] The Chinese character conversion dictionary data 17′ is used forconverting ‘hiragana’ characters into Chinese characters when a Chinesecharacter conversion mode is selected with a Chinese characterconversion button 36′. The Japanese dictionary data 18′ and theJapanese-English dictionary data 19′ are used for indicating the meaningof a word/phrase formed by the displayed adapted characters, and forindicating the English translation thereof, respectively, as shown inFIGS. 9 and 10.

[0119] In the following, the operation of the handwriting conversionapparatus according to the first embodiment of the present invention isdescribed with reference to the flow chart shown in FIG. 13.

[0120] Upon the initiation of the handwriting information processingprogram 9′ (step S1′), the input region 26′ is displayed on the displayapparatus 2 ₂, as shown in FIG. 4. When the user inputs handwritinginformation to the input region 26′ with an input pen (step S2′), theinput is detected and the input information is sent to the computer 1having input stroke detection functions, thereby initiating an inputstroke detection process.

[0121] Next, the input stroke detection process is described in furtherdetail with reference to FIG. 14.

[0122] As shown in FIG. 14, the computer 1 having input stroke detectionfunctions detects the input stroke 25′ (the path displayed by an inputoperation) of the input handwritten information (step S3′).Specifically, the computer 1 constantly monitors the process to see if ahandwriting input on the input region 26′ has been detected (step S20′),and when a handwriting input is detected, the coordinates of the inputpoints are sent to the coordinate value memory 10′ (step S21′). In thishandwriting information processing apparatus, the computer 1 detects thehandwriting input on the input region 26′ which is made by shielding(blocking the beam of) the laser emitted along the surface of thehandwriting input screen 2. The input coordinates of the handwritinginput are detected at predetermined time intervals (approximately 10msec) by the computer 1, and all the coordinates of the detected inputpoints 22′-24′ are sent to the coordinate value memory 10′ (see alsoFIG. 5). The coordinate value memory 10′ stores the received inputcoordinates.

[0123] Then the computer 1 determines whether or not the input has beencompleted (step S22′), and if the input has not been completed, thedetection of the input is continued. When the input has been completed,the input stroke 25′ is determined (step S23′), and a handwritinginformation determination process is initiated. The input completion ofthe handwritten information input is detected upon the termination ofthe shielding of the laser emitted along the surface of the handwritinginput screen 2.

[0124] The input stroke 25′ includes the input starting point 22′ atwhich the input is initiated, the partway input points 23′ that isdetected in the middle of the stroke, and the input ending point 24′ atwhich the input is terminated, as one stroke, and its movement and shapeare identified and determined by the computer 1. The input stroke 25′ isdisplayed on the handwriting input screen upon its input.

[0125] Now, the handwriting information determination process isdescribed in further detail with reference to FIG. 15.

[0126] After the determination of the input stroke 25′, the computer 1having handwriting information determination functions determineswhether the above determined input stroke 25′ is a handwriting characterinput or a editing operation input (step S4′).

[0127]FIG. 15 shows the specific procedures for the above handwritinginformation determination process.

[0128] First, information on the character line block region 21′ of thecharacter line block 20′ displayed on the display apparatus 2 ₂ isobtained from the character line block region memory 13′ (step S30′).This character line block region 21′ is determined by the coordinates(xa, ya) of a top left-hand corner point A and the coordinates (xb, yb)of a bottom right-hand corner point B, as shown in FIG. 7.

[0129] Next, each of the coordinates of the input points 22′-24′ of theinput stroke 25′ and the character line block region 21′ are compared(step S31′). Then it is determined whether or not any one of thecoordinates of the input points 22′-24′ is within the character lineblock region 21′ (step S32′). If at least one coordinate point isdetermined to be in the character line block region 21′, it isdetermined that the input stroke 25′ is an input stroke of an editingoperation input (step S34′) and an editing operation type determinationprocess is initiated.

[0130] For example, when a character line block 20″ made up of adaptedcharacters 101′-103′ (“

”) is displayed on the display apparatus 2 ₂, as shown in FIG. 16, inputstroke 40′ is determined to be an input stroke of an editing operationinput since one of its input points 40 a′ is in the character line blockregion of the character line block 20″. On the other hand, input stroke41′ is determined to be an input stroke of a handwritten character inputsince none of its input points are situated in the character line blockregion of the character line block 20″.

[0131] Also, in the above handwriting information determination process,if no adapted characters are displayed on the display apparatus 2 ₂, theinput stroke 25′ is determined to be an input stroke of a handwrittencharacter input.

[0132] In the following, the character recognition initiationdetermination process is described in further detail with reference toFIG. 17.

[0133] When the input stroke 25′ is determined to be an input stroke ofa handwritten character input, the computer 1 having characterrecognition initiation determination functions determines whether or notthe character recognition should be initiated (step S5′). The characterrecognition initiation determination process is performed according tothe specific procedures shown in FIG. 17.

[0134] In FIG. 17, the computer 1 determines whether or not a next inputhas been initiated (step S40′). If a next input is detected, the processgoes back to the input stroke detection process and new input strokedetection is initiated. If a next input is not detected, it isdetermined whether or not a prerequisite time has elapsed (step S41′),and if no input is detected before the elapse of the above prerequisitetime, the character recognition process is initiated.

[0135] The above prerequisite time can be changed according to thepreferences of the user; however, in this handwriting informationprocessing apparatus, the prerequisite time is pre-set to 1.5 sec.

[0136] Next, in the following, the character recognition process isdescribed in further detail with reference to FIG. 18.

[0137] With the determination to initiate the character recognition ofthe handwritten characters, the computer 1 having character recognitionprocessing functions initiates the character recognition process (stepS6′). The specific procedures for the above character recognitionprocess are described below with reference to FIG. 18.

[0138] First; the coordinate values of each of the input points in theinput stroke 25′ are obtained from the coordinate value memory 10′. Whena plurality of input strokes exists, the coordinate values of the inputpoints in all of these input strokes are obtained. Based on thecoordinates of each input point, a character cut-out process (step S51′)and a character recognition process (step S52′) are performed.

[0139] The above character cut-out process is a process of subdividingthe plurality of input strokes of a handwritten character input intosets of input strokes corresponding to a character. The characterrecognition process is a process of selecting character code data thatis closest to the input handwritten character using a well-knowncharacter recognition engine (character recognition processing program).These processes are both well-known in this technical field andtherefore, their descriptions are omitted.

[0140] In the handwriting information processing apparatus according tothis embodiment, the character code data that is closest to theoriginally input handwritten character is selected (selected charactercode data 11 a′), and character code data of candidate characters thatare quite similar in shape to the above handwritten characters and havebeen considered as candidates for the above selection are also extracted(candidate character code data 11 b′) The selected character code data11 a′ and the candidate character code data 11 b′ are sent to therecognition-adaptation result memory 11′ as results from the characterrecognition process (step S53′). In this manner, the characterrecognition process is successively performed for each of thehandwritten characters, and when the character recognition process iscompleted for all the handwritten characters (step S54′), the characteradaptation process is initiated.

[0141] Now, the character adaptation process will be described belowwith reference to FIG. 19.

[0142] With the termination of the character recognition process, thecomputer 1 having character adaptation processing functions initiatesthe character recognition process (step S7′). The specific proceduresfor the above character adaptation process are described below withreference to FIG. 19.

[0143] First, the coordinates of each of the input points in the inputstroke 25′ are obtained from the coordinate value memory 10′ (stepS60′). When a plurality of input strokes exist, the coordinates of eachof the input points in all of the above input strokes are obtained.

[0144] Then, based on the coordinates of each of the input points in theinput stroke 25′ of a handwritten character input, a handwrittencharacter block region 46′, indicated by dotted lines in FIG. 20, iscalculated and established (step S61′). As shown in FIG. 20, whenhandwritten characters 121′-123′ corresponding to the ‘hiragana’characters “

” are input to the input region 26′, coordinates (xp, yp) of a basepoint P and the height h of the handwritten character block region 46′for the above handwritten characters 121′-123′ (“

” are calculated, the handwritten character block region 46′ beingestablished based on the coordinates of the uppermost point 42′, thelowermost point 43′, and the leftmost point 44′ of the input strokes ofthe foremost character 121′ (“

”) among the handwritten characters 121′-123′.

[0145] The above base point P functions as a reference position fordisplaying the adapted characters 101′-103′ corresponding to thehandwritten characters, and the above-obtained height h is a referencefor setting the font size of the adapted characters. Specifically, whenthe font height of the adapted characters is denoted as h and thehorizontal to vertical ratio of the above font is denoted as t, thereference coordinate position in displaying the n^(th) adapted charactercounting from the foremost character is: (xp+(n−1)×t×h, yp).

[0146] In accordance with the above calculation method, the font size ofthe adapted characters corresponding to the handwritten characters isdetermined (step S63′) as well as their display positions (step S64′).The information on the font size and display position of the adaptedcharacters is sent to the recognition-adaptation result memory 11′ asfont size 11 g′ and position information 11 e′.

[0147] In this embodiment, the case in which the handwritten characters121′-123′ are written in a horizontal direction is described; however,it is also-possible to have the handwritten, character 121′-123′ writtenvertically. In the latter case, coordinates (xp, yp) of the base point Pand the height h of the handwritten character block region 46′ for theabove handwritten characters 121′-123′ (“

”) are calculated based on the coordinates of the uppermost point 42′,the leftmost point 44′, and the rightmost point 45′ of the input strokesof the foremost character 121′ (“

”)

[0148] Upon the determination of the font size and the display positionof the adapted characters corresponding to the handwritten characters,the adapted characters having the above determined properties aredisplayed on the display apparatus 2 ₂.

[0149] In the above, the configuration of the handwriting informationapparatus according to the first embodiment has been described. In thefollowing, the editing operation of the above handwriting informationapparatus will be described.

[0150] When the input stroke 25′ is determined to be an input stroke ofan editing operation input, the computer 1 having edit process typedetermination processing functions determines the type of editingprocess corresponding to the editing operation input (step S9′). Thespecific procedures for this edit process type determination process aredescribed below with reference to FIG. 21.

[0151] First, it is determined whether or not the input starting point22′ of the input stroke 25′ is situated within the character line blockregion 21′ (step S70′), and if the input starting point 22′ is not inthe character line block region 21′, it is determined whether or not theinput starting point 22′ is on a character line block edit pointer 47′(step S71′).

[0152] Herein, the character line block edit pointer 47′ is the smallsquare-shaped area assigned to each character line block and displayedon the display apparatus 2 ₂ for performing an editing process on theentire character line block 20′ (“

”), as shown in FIG. 22. Also, on the display apparatus 2 ₂, aone-character edit pointer 48′, which is a small square-shaped pointerassigned to each character for performing an editing process on eachcharacter, is provided. In the present embodiment, the character lineblock edit pointer 47′ is provided outside the character line blockregion 21′, and the one-character edit pointer 48′ is provided withinthe character line block region 21′ for each of the characters. Theseediting pointers may be displayed at the same time the adaptedcharacters 101′-103′ (“

”) are displayed on the display apparatus 2 ₂, or they may be displayedafter the character line block 21′ is selected with an input pen.

[0153] When the input starting point 22′ is on the character line blockedit pointer 47′, it is determined that the editing operation input isfor performing an editing process of changing the font size for theentire character line block 20′, thereby initiating the character lineblock editing process. On the other hand, when the input starting point22′ is not on the character line block edit pointer 47′, it isdetermined that the editing operation input is for performing a gestureediting process, thereby initiating the gesture editing process.

[0154] When the input starting point 22′ is within the character lineblock region 21′, it is further determined whether or not the inputstarting point 22′ is on the one-character edit pointer 48′ (step S72′).When the input starting point 22′ is on the one-character edit pointer48′, it is determined that the editing process is for one character,thereby initiating a one-character editing process. When the inputstarting point 22′ is not on the one-character edit pointer 48′, it isdetermined that the editing process is for the entire character lineblock 20′, thereby initiating a character line block editing process.

[0155] In the following, the one-character editing process is describedin further detail with reference to FIG. 23.

[0156] When the input starting point 22′ is on the one-character editpointer 48′, it is determined that the one-character edit pointer 48′has been selected, and the computer 1 having one-character editprocessing functions initiates the one-character editing process (stepS10′) The specific procedures of the above one-character editing processare described below with reference to FIG. 23.

[0157] First, with the selection of the one-character edit pointer 48′,an editing panel 49′ is displayed on the display apparatus 2 ₂ (stepS80′). For example, referring to FIG. 24, if the user wishes to edit theadapted character 102″ (“

”) in the character line block 20′ having the adapted characters 101′,102″, and 103′ (“

”), the user may select the one-character edit pointer 48′ correspondingto the adapted character 102″ (“

”) and in response, the editing panel 49′ is displayed in the vicinityof the adapted character (“

”). At the same time, the-adapted character 102″(“

”) is highlighted on the display apparatus 2 ₂.

[0158] The above editing panel 49′ includes a delete button 50′, aninsert button 51′, candidate character buttons 52′, and an escape button53′. An ID is pre-assigned to each of the above buttons. That is, thedelete button 50′ has a process of deleting a character assignedthereto, the insert button 51′ has a process of inserting a spaceassigned thereto, and the candidate character button 52′ has a processof replacing a character with a different candidate character assignedthereto. Data on the above one-character editing processes of theediting panel 49′ are linked to their respective button IDs and storedin the edit memory 12′.

[0159] Upon the selection of one of the buttons within the editing panel49′ (step S81′), the button ID of the selected button is obtained (stepS82′), and an editing process corresponding to the above-obtained buttonID is performed (step S83′).

[0160] For example, when the delete button 50′ is selected, the adaptedcharacter 102′ (“

”) is deleted, as shown in FIG. 25A, and when the insert button 51′ isselected, a space 125′ is inserted between the adapted character 101′ (“

”) and the adapted character 102″ (“

”), as shown in FIG. 25B. Also, when a button corresponding to the codecharacter “

” among the candidate character buttons 52′ is selected, the adaptedcharacter 102″ (“

”) is replaced with the adapted character 102′ (“

”), thereby obtaining the character line block 20′ with the adaptedcharacters 101′, 102′, and 103′ (“

”), as shown in FIG. 25C.

[0161] Further, if a new handwritten character 104′ (“

”) is input to the position where the adapted character 102″(“

”) is placed, at the time the editing panel 49′ is displayed, the newhandwritten character (“

”) is recognized and adapted via the respective processes and theadapted character 102″ (“

”) is deleted to obtain the adapted characters 101′, 104′, and 103′ (“

”) in the character line block 20′, as shown in FIG. 25D. The escapebutton is a button for canceling the one-character editing process, andthe editing panel 49′ disappears upon pressing this button.

[0162] When the one-character editing process is performed, the editingresult is reflected and displayed on the display apparatus 2 ₂.

[0163] Next, the character line block editing process is described withreference to FIG. 26.

[0164] As shown in FIG. 26, when the computer 1 having character lineblock edit processing functions initiates the character line blockediting process (step S11′), it is first determined whether or not theinput starting point 22′ of the input stroke 25′ is on the characterline block edit pointer 47′ (step S90′) If the input starting point 22′is on the character line block edit pointer 47′, it is determined thatthe editing process is a font size changing process for the entirecharacter line block (step S91′).

[0165] The above font size changing process is a process of changing thefont size of the adapted characters in the character line block 20′based on the moving direction and the moving distance of the inputstroke 25′ passing through the input points from the input startingpoint 22′ to the input ending point 24′. For example, as shown in FIG.27, when the input stroke 25′ starting at the character line block editpointer 47′ of the character line block 20′ having the adaptedcharacters 101′-103′ (“

”) moves diagonally in an lower-right direction to end at the inputending point 24′, the font size of the adapted characters 101′-103′ (“

”) in the character line block 20′ is enlarged with base point P as thepivot.

[0166] In this case, the font size of the adapted characters 101′-103′(“

”) in the character line block 20′ may be changed while maintaining thehorizontal-to-vertical ratio of the character line block 20′, by makingthe change based on only one of either the moving distance in theleft-to-right direction or the moving distance in the up-down directionof the input stroke 25′. Alternatively, the font size of the adaptedcharacters 101′-103′ (“

”) in the character line block 20′ may be changed by considering boththe moving distance in the left-to-right direction and the movingdistance in the up-down direction of the input stroke 25′ so that thehorizontal-to-vertical ratio of the character line block 20′ is alsochanged.

[0167] In the case where the input starting point 22′ is not on thecharacter line block edit pointer 47′, it is determined whether or notthe number of repetitive movements in a predetermined direction made bythe input stroke 25′ exceeds a predetermined number of times (stepS92′). If the number of repetitive movements in a predetermineddirection made by the input stroke 25′ exceeds the predetermined numberof times, the character translation process is performed (step S93′).

[0168] The above character translation process is a process ofindicating a translation of a word/phrase represented by the charactersin the character line block 20′. For example, in FIG. 9, adaptedcharacters 106′ and 107′ making up the word “

” are shown in the character line block 20′, and when the input stroke25′ makes repetitive movements in a left-to-right direction for apredetermined number of times in the character line block region 21′,reference is made to the Japanese-English dictionary data 19′ in thehard disk 8′ and an English translation 56 (“Translation”) of the word “

” is displayed on the display apparatus 2 ₂.

[0169] Alternatively, in a case where alphabets are input to the inputregion 26′, and are recognized and adapted into adapted characters ofthe alphabet, and an English word/phrase formed by the characters isdisplayed in the character line block 20′, the above-described charactertranslation process may be a process of referring to theEnglish-Japanese dictionary data and indicating a Japanese translationof the above English word/phrase. Also, as shown in FIG. 10, the abovecharacter translation process may be a process of indicating the meaningof the word formed by the adapted characters 106′ and 107′ (“

”) shown in the character line block 20′ using the Japanese dictionarydata.

[0170] As the criteria for starting the character translation process,the predetermined number of repetitive movements may be set to 3-6times, for example, so that it can be ascertained that the aboverepetitive movement is an intentional move by the user but therepetition will not impose a significant burden. In this embodiment, thedirection of the repetitive movement is in a left-to-right direction;however, this can also be in an up-down direction, or differentcharacter translation processes may be assigned to each of theleft-to-right direction movement and the up-down direction movement,respectively.

[0171] Additionally, other various processes may be assigned todifferent repetitive movements of over a predetermined number of timesin a predetermined direction made by the input stroke 25′ such as aprocess of displaying ‘rubi’ characters or a process of displaying anexplanation of a coined term, and the like.

[0172] If the number of repetitive movements of the input stroke 25′ isbelow the predetermined number of times, it is determined whether or notthe moving speed of the input stroke 25′ is greater than a predeterminedspeed (step S94′), and if the moving speed of the input stroke 25′ isgreater than a predetermined speed, the character alignment process isperformed (step S95′) This character alignment process is a process ofaligning the adapted characters to the edge of the display apparatus 2 ₂according to the moving direction of the input stroke 25′.

[0173] For example, in FIG. 8A, the character line block 20′ containingthe characters “

” is displayed around the center of the display apparatus 2 ₂, and whenthe input stroke 25′ is input from within the character line blockregion 21′ to the left at a faster speed than the predetermined speed,the character line block 20′ is indented to the left side of the inputregion 26′ of the display apparatus 2 ₂, as shown in FIG. 8B. At thesame time, the adapted characters in the character line block 20′ arerearranged closer together for a more presentable appearance.

[0174] When the moving direction of the above input stroke 25′ is to theright, the character line block 20′ is indented to the right side of theinput region 26′ of the display apparatus 2 ₂, as shown in FIG. 8C. Atthe same time, the adapted characters in the character line block 20′are assigned equal spacing for a more presentable appearance.Alternatively, when the input stroke 25′ moves up or down, the characterline block 20′ can be moved to the upper edge or bottom edge of theinput region 26′, respectively.

[0175] When the moving speed of the input stroke 25′ is below thepredetermined speed, it is determined whether or not the input endingpoint 24′ of the input stroke 25′ is in the region of another characterline block (step S96′). If the input ending point 24′ of the inputstroke 25′ is in the region of another character line block, thecharacter property association process is performed.

[0176] The above character property association process is the processof associating the properties of the adapted characters in differentcharacter line blocks that are displayed on the display apparatus 2 ₂.

[0177] For example, in FIG. 11, the input starting point 22′ of theinput stroke 25′ is within the character line block region 21′ of thecharacter line block 20′, and the input ending point 24′ is in thecharacter line block region 39″ of the character line block 39′. Thus,the font size and the character spacing of the adapted characters “

” in the character line block 20′ are associated with and made toconform to the font size and character spacing of the adapted character“

” in the character line block 39′, and the adapted characters “

” and “

” are displayed in the same font size and with the same characterspacing.

[0178] In the present embodiment, the character property associationprocess corresponds to the process of causing the conforming of the fontsize and character spacing; however, this may be a process of causingthe conforming of the font type, font color, or other characterproperties. Also, this process is not limited to causing the conformingof the properties; thus, in the above example, the adapted characters inthe character line block 20′ may be arranged to have a font size that ishalf the size of the font of the adapted characters in the characterline block 39′.

[0179] If the input ending point 24′ of the input stroke 25′ is not inthe region of another character line block, the moving process isperformed. This moving process corresponds to a process of moving theentire character line block to the moving direction of the input stroke25′.

[0180] For example, with reference to FIG. 28, when the input stroke 25′starts from within the character line block region 21′ and ends at apredetermined point in the input region 26′ that is outside of anothercharacter line block region, the character line block 20′ is moved alongthe path of the input stroke 25′ from the input starting point 22′ tothe input ending point 24′.

[0181] When the above character line block editing process is performed,the editing results are reflected and displayed on the display apparatus2 ₂.

[0182] In the following, the gesture editing process is described infurther detail with reference to FIGS. 12 and 29.

[0183] When the input starting point 22′ of the input stroke is neitherin the character line block region 21′ nor on the character line blockedit pointer 47′, the computer 1 having gesture edit processingfunctions initiates the gesture editing process (step S12′).

[0184] The above gesture editing process is for performing variousediting processes on the adapted characters based on the input form ofthe input stroke 25′. As mentioned earlier, if none of the input pointsof the input stroke 25′ from the input starting point 22′ to the inputending point 24′ is within the character line block region 21′, thenthis input stroke 25′ is determined to be a handwritten character input.Thus, in order for the gesture editing process to be initiated, at leastone input point of the input stroke 25′ needs to be in the characterline block region 21′. For example, an input stroke 25′ that starts fromoutside the character line block region 21′ and ends at a point in thecharacter line block region 21′, or an input stroke 25′ that starts fromoutside the character line block region 21′, enters the character lineblock region 21′, and ends at a point outside the character line blockregion 21′ both fit into the above case.

[0185] As shown in FIG. 29, when the gesture editing process isinitiated, first of all, the shape of the input stroke is identified(step S100′). In the gesture edit memory 15′ of the hard disk 8′, dataon the gesture editing processes assigned to the respective shapes(forms) of the input stroke are stored. For example, an input stroke inthe form of slashing across the characters from right to leftcorresponds to a deletion process on the slashed characters. Also, aninput stroke in the form of encircling a character corresponds to aprocess of changing the font color of the circled character.

[0186] The input form of the input stroke 25′ is identified based on theinformation stored in the gesture edit memory 15′; in turn, the gestureediting process is determined according to the form of the input stroke25′ (step S101′). At the same time, the character on which the editingprocess is performed is also determined based on the form of the inputstroke 25′ (step S102′).

[0187] For example, as shown in FIG. 12A, when a gesture input 54′,which is the input stroke 25′ in the form of slashing across the adaptedcharacters 111′ and 112′ (“

”) in a right-to-left direction, is made on the character line block 20′made up of the adapted characters 108′-112′ (“

”), the adapted characters 111′ and 112′ (“

”) are deleted by the gesture editing process and the displayedcharacters in the character line block 20′ will be the adaptedcharacters 108′-110′ (“

”).

[0188] Also, as shown in FIG. 12B, when a gesture input 55′ ofencircling the adapted characters 110′ and 111′ (“

”) in the character line block 20′ made up of the adapted characters108′-112′ (“

”) is made, the font color of the adapted characters 110′ and 111′ (“

”) will be changed by the gesture editing process.

[0189] In this way, the gesture editing process is performed on therespective character based on the input form of the input stroke 25′(step S103′). When the gesture editing process is performed, the editresults are reflected and displayed on the display apparatus 2 ₂.

[0190] The components of each of the one-character editing process, thecharacter line block editing process, and the gesture editing processand the allocation of the input operations to the respective editingprocesses are not limited to the above-described embodiment and variouschanges and modifications are possible as long as the components of eachof the editing processes and the respective editing operation inputs areallocated beforehand and stored in the panel edit memory 12′, thecharacter line block edit memory 14′, the gesture edit memory 15′, andthe like.

[0191] Now, referring back to FIG. 1, a description of a handwritinginformation processing apparatus according to a second embodiment of thepresent invention will be given.

[0192] According to this embodiment, the coordinate input apparatus 2 ₁of FIG. 1 is an apparatus that detects a signal input thereto as acoordinate value, the signal being input by an operator (user) 3 usingan indication part 3 ₁ such as a finger or a pen. The coordinate inputapparatus 2 ₁ then sends this information to the computer 1. Thecomputer 1 stores all the information on the collective set ofcoordinate values (referred to as ‘stroke’ hereinafter) sent from thecoordinate input apparatus 2 ₁, performs character recognition in abatch upon receiving instructions to perform the handwritten characterrecognition from the user (operator) 3, and displays the obtainedprocessing results on the display apparatus 2 ₂.

[0193]FIG. 30 is a flow chart of a handwriting information processingapparatus according to the second embodiment of the present invention.

[0194] Referring to FIG. 30, the coordinate input apparatus 2 ₁ and thedisplay apparatus 2 ₂ make up the handwriting input/display apparatus 2shown in FIG. 1, and the process illustrated in between the two blocks 2₁ and 2 ₂ in FIG. 30 is performed in the computer 1 (referred to as ‘PC1’ hereinafter).

[0195] First, in a recognition instruction discrimination process, it isdetermined whether or not an instruction to perform a recognitionprocess on the coordinate values output from the coordinate inputapparatus 2 ₁ is sent by the user (step S10) When the recognitioninstruction is not detected (step S10-No), the coordinate values inputto the handwriting input region are stored in the coordinate valuememory 101 by a coordinate value storing process (step S20) after whichthe process goes back to step S10 where the monitoring for a recognitioninstruction from the user is continued. When a recognition instructionis detected (step S10-Yes), the handwritten character recognitionprocess is performed by referring to the coordinate values memory 101,and character strings obtained from this process are stored in theresult memory 102 (step S30).

[0196] Then, an IME (Input Method Editor) Chinese character conversionis performed by referring to the result memory 102, wherein thecharacter strings obtained from the handwritten character recognitionprocess are converted into Chinese characters, and the results of thisprocess are stored in the result memory 102 (step S40).

[0197] Further, an outline drafting process is performed whereininformation on the positions in which the handwritten characters arewritten is obtained by referring to the coordinate values memory 101,and an indent position is calculated from the positions of the characterstrings and stored in the result memory 102 (step S50).

[0198] Next, a display process is performed by referring to the resultmemory 102, wherein the results of the process are output to the displayapparatus 2 ₂ (step S60), and at the same time, a user operationdetermination process is initiated (step S70). The user operationdetermination process of step S70 is a process of determining thereaction of the user to the results displayed on the display apparatus 2₂ in the display process of step S60. After the user operationdetermination process of step S70, the process in the PC 1 can go in twodifferent directions. In one case the user edits the displayed results(step S70-Edit). For example, the editing can be a correction of theindent position obtained from the outline drafting process of step S50.Thereby, in step S80, an editing operation is performed within anediting process by referring to the result memory 102, and the resultsare output to the display process of step S60. In the other case, theuser outputs the displayed results (step S70-Output) The outputting ofthe results is performed within an outputting process by referring tothe result memory 102, wherein the data organized in a predeterminedformat are output to a storage medium 4 such as a hard disk or anoptical disk, or the data with an address and a title entered theretoare output to a network 5 as mail for transmission and the like (stepS90) After the outputting process of step S90 is completed, the processgoes back to the recognition instruction discrimination process of step10, so that handwritten characters can be newly input to the handwritinginput/display apparatus 2.

[0199] In the following, each of the steps shown in FIG. 30 is describedin further detail.

[0200] First, the recognition instruction discrimination process will bedescribed.

[0201]FIG. 31 shows an example of the state of the handwriting inputscreen of the display apparatus 2 ₂ when the recognition instructiondiscrimination process is to be performed.

[0202] As shown in FIG. 31, the handwriting input screen 2 is prepared,wherein a part of the handwriting input screen 2 surrounded by a dottedline is the region in which handwriting is input (handwriting inputregion 2 a), and a recognition button is provided above the handwritinginput region 2 a. By monitoring the position of the user's input to thehandwriting input screen 2, it is possible to determine whether or notthe user has initiated a recognition process. The output of therecognition instruction discrimination process of step S10 can go in twodifferent directions. In one case, the user inputs handwrittencharacters to the handwriting input region 2 a, and the process movesonto step S20 wherein a coordinate value storing process is performed.In the other case, the user pushes the recognition button 2 b at adiscretionary timing, and the process moves onto step S30 wherein thehandwritten character recognition process is performed.

[0203] Next, the coordinate value storing process, which is the firstone of the output destinations for the recognition instructiondiscrimination process, will be described.

[0204]FIG. 32 is a flow chart for illustrating the coordinate valuestoring process.

[0205] First, as shown in FIG. 32, the coordinate values of thehandwritten characters input to the handwriting input region 2 a by theuser are subdivided into each stroke unit by a stroke subdividingprocess, the results therefrom being stored in the coordinate valuememory 101 (step S21). For example, the user's touching the coordinateinput apparatus 2 ₁ of the handwriting input/display apparatus 2corresponds to clicking the left button of a mouse. In an operatingsystem such as Windows (registered trademark), the messages MouseDown,MouseMove, MouseUp are assigned to the left-click operation. Thecoordinate input apparatus 2 ₁ of the handwriting input/displayapparatus 2 instructs the operating system to generate these messagesfor each detection of a coordinate value. Note that the above-describedstroke is a collection of coordinate values detected from a MouseDownpoint to a MouseUp point.

[0206] Next, a line subdivision process is performed wherein an inquiryis made to determine which of the strokes belong to one particular line,the results of this process being stored in the coordinate value memory101 (step S22).

[0207] Finally, the process is moved back to the recognition instructiondiscrimination process of step S10. Thereby, the coordinate valuestoring process of step S20 runs in a loop until a recognitioninstruction is made by the user.

[0208]FIG. 33 is a diagram for illustrating the stroke subdivisionprocess.

[0209] The stroke subdivision process of step 21 included in theabove-described coordinate value storing process of step S20 is aprocess of storing a collective set of coordinate values belonging toone stroke of a character in the coordinate value memory 101, theabove-described one stroke being the coordinate values detected from thepoint where the user touches the handwriting input region 2 a with theindicating part 3 ₁ to move it along the surface of the handwritinginput region 2 a until the point where the user leaves off, as shown inFIG. 33.

[0210]FIG. 34 is a diagram for illustrating the line subdivisionprocess.

[0211] For example, it is supposed that in the first line of thehandwriting input region 2 a a character string “

” is written, and in the second line, a character string “

” is written, as shown in FIG. 34. When these character strings aresubdivided into their respective lines, the character string “

” in the first line contains the first to seventh strokes, and thecharacter string “

” in the second line contains the eighth stroke to the fifteenth stroke.

[0212]FIG. 35 is a diagram for illustrating the operation of the linesubdivision process.

[0213] The above-described line subdivision process of step S22 isperformed in the following manner. First, a stroke 1 is drawn in thehandwriting input region 2 a, and a maximum value ymax and a minimumvalue ymin of the y coordinates corresponding to the coordinate valuesthat belong to the stroke 1 are obtained, as shown in FIG. 35A. Then, asecond stroke 2 is drawn, and when there is at least one coordinatepoint belonging to the stroke 2 that has a y coordinate value that iswithin the maximum value ymax and the minimum value ymin, as shown inFIG. 35B, the stroke 2 is regarded as being in the same line as stroke1. Consequently, the maximum value ymax and the minimum value ymin arerecalculated, as shown in FIG. 35B′ and information on the line isstored in the coordinate value memory 101. On the other hand, when thereis no point belonging to the stroke 2 that has a y coordinate value thatis within the maximum value ymax and the minimum value ymin, as shown inFIG. 35C, the stroke 2 is regarded as the beginning of a new line.Consequently, the maximum value ymax and the minimum value ymin arenewly calculated for the stroke 2, as shown in FIG. 35C′ and a new linenumber is stored in the coordinate value memory 101.

[0214]FIG. 36 is a diagram showing an example of the coordinate value,the stroke information, and the line information stored in thecoordinate value memory as a result of the coordinate value storingprocess.

[0215] The information to be stored in the coordinate value memory 101as a result of the process of step S21 and step S22 includes acoordinate value (x, y), the stroke number, and a line number for eachcoordinate point, as shown in FIG. 36.

[0216] Next, the handwritten character recognition process, which is thesecond one of the output destinations for the recognition instructiondiscrimination process, will be described.

[0217]FIG. 37 is a flow chart illustrating the handwritten characterrecognition process, which is described in detail below.

[0218] First, referring to the coordinate value memory 101, thecoordinate values of one line are called (step S31). Then, a process ofcutting out each one of the characters is performed on the coordinatevalues that have been called (step S32). There are various methods forthe above process of cutting out each one of the characters, and anymethod can be used for this process. The process of cutting out each oneof the characters is performed in step S32 in order to determine whichof the strokes constitute a character.

[0219] Next, a determination of whether or not there is a character tobe recognized is made (step S33), and when a character to be recognizeddoes exist (step S33-Yes), then the coordinate values cut out for eachof the characters are sent to a recognition engine. The character codesrecognized in the recognition engine are stored in the result memory 102for each line number (step S34). There are also numerous configurationsof the recognition engine, and again, any type of recognition engine canbe used in this embodiment.

[0220] Subsequently, the process is moved back to step S33 where it isdetermined whether or not a character to be recognized still exists, andwhen a character to be recognized does exist (step S33-Yes), this issent to the recognition engine. When a character that is to berecognized does not exist (step S33-No), it is determined whether or nota line to be determined still exists (step S35). When the line doesexist (step S35-Yes), the process goes back to step S31 wherein aprocess of calling the coordinate values of one line is performed andthe above-described processes in the steps S31-S35 are repeated. Whensuch a line does not exist (step S35-No), the process moves on to a IMEChinese character conversion process of step S40.

[0221]FIG. 38 is diagram showing an example of the information stored inthe result memory as a result of the handwritten character recognitionprocess.

[0222] The information stored in the result memory 102 as a result ofthe above-described processes of the steps S31-S35 is of the recognizedcharacter strings for each line. In the result memory 102, indentinformation including a line starting position, an indent number, aparent line number, and a subordinate line number, is also stored;however, the description thereof is given later in the description ofthe outline drafting process.

[0223] Next, a description of the IME Chinese character conversionprocess following the handwritten character recognition process isgiven.

[0224]FIG. 39 is a flow chart for illustrating the IME Chinese characterconversion process, which is described in detail below.

[0225] Upon receiving the process after the determination of whether ornot a next line exists in step S35 of the above-described handwrittencharacter recognition process of step S30, the recognized characterstring of one line is called by referring to the result memory 102 (stepS41). As shown in FIG. 38, the recognized character strings are storedin the result memory 102 for each line, and thus, the above callingprocess of step S41 can be easily performed.

[0226] The called recognized character string is sent to an IME (InputMethod Editor) such as MS-IME or ATOK. Consequently, the IME sends backthe Chinese character converted results of the recognized characterstring, so that this result can be written onto the recognized characterstring stored in the result memory 102 (step S42). Alternatively, theresult memory 102 may secure a region for storing the IME Chinesecharacter-converted character string.

[0227] Subsequently, it is determined whether or not a next line existsin the result memory 102 (step S43), and when a next line does exist(step S43-Yes), the process goes back to the process of calling therecognized character string of one line. When the next line does notexist (step S43-No), the process moves on to the outline draftingprocess of step S50.

[0228] In this way, handwriting information can be stored as the strokeinformation and the line information and can be batch processed forrecognition at an arbitrary timing chosen by the user rather thanprogressively converting the handwritten characters into typed text.Also, the recognized character string can be sent to the IME for Chinesecharacter conversion, thereby simplifying the Chinese characterconversion process.

[0229] Next, the outline drafting process that follows the IME Chinesecharacter conversion process is described.

[0230]FIG. 40 is a flow chart for illustrating the outline draftingprocess, which is described in detail below.

[0231] First, in a line starting position calculation process, the linestarting position is calculated for each line by referring to thecoordinate value memory 101 (step S51), and the results are stored tothe result memory 102. Then, it is determined in the next step S52whether or not a next line exists, and if a next line does exist (stepS52-Yes), the process goes back to step S51 wherein the line startingposition calculation process is performed so as to determine thecoordinate value of the starting position of each line. Specifically, bydetermining in each of the lines the minimum values of the x coordinateand the y coordinate of the coordinates points belonging to a respectiveline, the top-left coordinates of the line are obtained and this isregarded as the line starting position. When a next line is not found(step S52-No), the process is moved to the next step.

[0232] Next, for each of the line starting positions obtained from theabove line starting position calculation process, the x coordinate of aline is compared with that of the line above, starting with the linewith the smallest y coordinate value. Thus, a process of indenting thelines is performed, the results being stored in the result memory 102(step S53). The above process is performed according to the order of they coordinate values of the line, starting with the line with thesmallest y coordinate value, because the user will not necessarily inputcharacter strings in the order of the y value coordinates. For example,in a case where a first, second, and third line are written andafterwards a fourth line is inserted in between the first and secondlines, the appropriate indent position cannot be obtained if the aboveprocess is performed in the order, of the line numbers. Also, ‘the lineabove’ is referred to as the line having a y coordinate valuecorresponding to its starting position that is next in order withrespect to the y coordinate value of the starting position of the linebeing processed.

[0233] Subsequently, it is determined whether or not a next line exists(step S54), and when a next line to forgo the above indenting processdoes exist (step S54-Yes), the process goes back to step S53 and theindenting process is repeated. When the next line does not exist (stepS54-No), the process moves on to the display process of step S60.

[0234] In the following, the line starting position calculation processis described in further detail.

[0235]FIG. 41 is a diagram for illustrating the line starting positioncalculation process.

[0236] For example, in a case where four lines of character strings arewritten in the handwriting input region 2 a, as shown in FIG. 41, thestarting positions of the lines are obtained as the coordinate values(x1, x2) for line 1, (x2, y2) for line 2, (x3, y3) for line 3, and (x4,y4) for line 4. These coordinate values are stored as the line startingposition in the line information of each line in the result memory 102.

[0237] Next, in the following, the indenting process is described infurther detail.

[0238]FIG. 42 is a diagram for illustrating the indenting process.

[0239] The indenting process is a process of setting the line numbercorresponding to the indent number, the parent line number, and thesubordinate line number of each line stored in the result memory 102.There is only one value for the indent number and the parent number,respectively, but the subordinate number may possibly have a pluralityof values.

[0240] First, the x coordinate value of the line with the characterstring “

” written thereon in the handwriting region 2 a is denoted as xp and thex coordinate value of the line above having the character string “

” written thereon is denoted as xq, and a value S is obtained fromsubtracting xq from xp.

[0241] If S is greater than W (i.e. the indent width), as shown in FIG.42A, then the line above will be the parent line and the line to beprocesses will be the subordinate line. If S is greater than −W and lessthan W (−W<S<W), as shown in FIG. 42B, then the line to be processed andthe line above are in equivalent positions. If S is less than −W, asshown in FIG. 42C, the parent line number is traced back for the valueof a quotient Q obtained from dividing S by −W with respect to theparent line number of the line situated right above. It is noted thatherein, W represents the indent width, and this value can be preset bythe user.

[0242] As a specific example, the indenting process performed on thediagram shown in FIG. 41 is described.

[0243] Upon searching for the line with the smallest y coordinate value,line 1 with a y coordinate value of y1 is detected and this becomes thefirst line to be processed. Since there is no line above line 1, theindent number of this line is set to “0”, the parent line number to “0”,and no value is set for the subordinate line number. Upon searching forthe next smallest y coordinate value, line 4 is detected and thisbecomes the next line to be processed. Thus, the line above line 4 isline 1. Here, the value S4, obtained by subtracting the x coordinatevalue x1 of line 1 from the x coordinate value x4 of line 4, is greaterthan the indent width W. Thereby, line 1 will be the parent line andline 4, the subordinate line. In this process, “4” is added to thesubordinate line number item in the line information of line 1; theindent number of line 4 which is the value obtained from adding 1 to theindent number of line 1 is set to “1”; and the parent line number ofline 4 is set to “1”. Then, upon searching for the next smallest ycoordinate value, line two becomes the line to be processed, and line 4becomes the line above line 2. As in the above description, the valueS2=x2−x4 is obtained, and in this case the value of S2 is greater than−W and less than W. Thus, line 4 and line 2 are placed in equivalentindent positions. Therefore, the indent number of line 2 is the same asthat of line 4; and the parent line number is also the same as that ofline 4, this being “1”.

[0244] Subsequently, upon searching for the next smallest y coordinatevalue, line 3 is determined to be the next line to be processed, and theline 2 becomes the line above line 3. Then as in the above process, thevalue S3=x3−x2 is obtained, and in this case, S3 is less than −W. Thus,line 3 is superordinate in relation to line 2. Since the quotient Qobtained by dividing S3 by −W is “1”, the parent line number of line 2is traced back, for the value of Q, namely, the parent number is tracedback to a value one rank higher with respect to that of line 2. In thiscase, since the parent line number of line 2 is “1”, the parent linenumber of line 1 is referred to, and since the indent number of line 1is “0” and its parent line number is “0”, the indent number of line 3 isset to “0” and its parent line number is also set to “0”. Since a nextline does not exist, the results obtained from the above-describedprocess are stored in the result memory 102, and then the process moveson to the display process of step S60.

[0245] Next, the display process following the outline drafting processis described in further detail.

[0246]FIG. 43 is a flow chart for illustrating the display process.

[0247] In the display process of step S60, the data stored in the resultmemory 102 are displayed on the display apparatus 2 ₂.

[0248] In this process, first, a textbox is made for each of the linesthrough a textbox making process (step S61). The textbox can be easilymade using programming tools such as Visual C++ or Visual Basic. Then ina text providing process, the recognized character strings stored in theresult memory 102 are provided as the text (step S62). Further, in atextbox display position calculation process, the display position ofthe textbox of line n is set to (W×In, yn), provided that Yn denotes they coordinate value of the line starting position of line n and Indenotes the indent number of line n (step S63).

[0249] Next, a determination of whether or not a next line on which theabove-described process is to be performed exists is made (step S64),and when there is a next line to be processed (step S64-Yes), theprocess goes back to step S61 and the above-described process isrepeated on all the line information stored in the result memory 102 andthe results displayed on the display apparatus 2 ₂. When a next linedoes not exist (step S64-No), the process moves on to the user operationdetermination process of step S70.

[0250] Next, the user operation determination process following thedisplay process is described in further detail.

[0251]FIG. 44 is a flow chart illustrating the user operationdetermination process; and FIG. 45 is an example of the handwritinginput screen when the user operation determination process is performed.

[0252] As for the user operation, there is the editing process forcorrecting the outlined information, and the outputting process forstoring the obtained information in a storage medium or sending this viae-mail. First, upon the initiation of the user operation determinationprocess, a button alteration process is performed wherein thepreviously-described “recognition”button 2 b of FIG. 31 disappears andan ”output” button 2 c is displayed, as shown in FIG. 45 (step S71).Then, in step S72, monitoring is performed to see whether the user hasinput something in the handwriting input region 2 a or has pushed the“output” button 2 c. In the case where an input has been made in thehandwriting input region 2 a (step S72-Edit), the process moves on tothe editing process of step S80. On the other hand, in a case where the“output” button 2 c has been pressed (step S72-Output), the processmoves on to the output process of step S90. In this way, the output ofthe user operation determination process of step S70 can go in twodifferent directions.

[0253] Next, the editing process, which is the first one of the outputdestinations of the user operation determination process is described infurther detail below.

[0254]FIG. 46 is a flow chart for illustrating the editing process.

[0255] In the editing process of step S80, the outline information suchas the indent position of each of the lines obtained in the outlinedrafting process is corrected according to the instructions made by theuser. In this process, if textboxes are made using a programming toolsuch as Visual C++ or Visual Basic, the MouseDown, MouseMove, MouseUpmessages can be monitored in the textboxes. Each of the textboxescorrespond to each of the respective lines stored in the result memory102, and when a user touches a textbox with the indicating part, aMouseDown message is received and the line that has been selected by theuser can be discerned. This in turn enables a selected linedetermination process of step S81.

[0256] Subsequently, thee user moves the indicating part around whilestill touching the textbox. This is received as a MouseMove message andthe amount of movement can be discerned from this action on the user'sbehalf. Based on the above-obtained movement amount, the displayedposition of the textbox can be changed and the user's operation ofmoving the textbox is realized. Then the user can let go of the textboxat any spot. At this point, the MouseUp message is received, and themoving destination of the textbox is determined, thereby enabling amoving destination determination process (step S82). When the movingdestination is determined in the moving destination determinationprocess, the x coordinate value of the gridline (to be described later)and the y coordinate value of the moving destination are written ontothe line starting position information of the moved line (the linedetermined in the selected line determination process) that is stored inthe result memory 102 (step S82). Then, an indent correction process isperformed at the moving destination, wherein the results are stored inthe result memory 102, and the process goes back to the display processof step S60 (step S83).

[0257]FIG. 47 is a diagram showing an example of the state of thehandwriting input region when the editing process is to be performed.

[0258] As described above, in the editing process of step S80, a movingoperation of the textbox is performed by the user. Upon this occasion,gridlines indicating the indent width can be provided in the handwritinginput region 2 a, as shown in FIG. 47, and it can be arranged so thatthe textboxes can only be moved onto the gridlines. In this way, thetextboxes can be precisely moved to the desired indent positions. Here,it is noted that the gridlines can be indicated or hidden depending onthe choice of the user.

[0259]FIGS. 48A and 48B are diagrams illustrating the indent correctionprocess.

[0260] In the following, a process of moving the line 3 shown in FIG. 47to a spot in between line 1 and line 2, as shown in FIG. 48A, isdescribed as an example.

[0261] The alteration of the indent number, the parent line number andthe subordinate line number in the result memory 102 can be performed inthe same manner as that described in the description of the outlinedrafting process. The line above line 3 is line 1, and since the line 3is moved to stand under line 1 (subordinate line), “3” is added to thesubordinate line number information of line 1, the indent number of line3 is set to the sum of the indent number of line 1 plus 1, which is “1”,and the parent line number of line 3 is set to “1”. There are threemajor differences between this process and the formal outline process.First, in this process, the subordinate line of line 2 needs to bechanged with the moving of line 3. This can be realized by referring tothe parent line number information of line 3 upon moving the line 3 andeliminating the “3” from the subordinate line number information of line2 when it is discerned that line 2 is the parent line of line 3. Second,in this process, “4” is included in the subordinate line information ofline 3. Thus, when moving the line 3, all the lines that come under theline 3 are moved along with line 3. Thereby, the line starting positioninformation of line 4 stored in the result memory 102 is altered basedon the amount of movement in line 3, and the indent number of line 4 isset to the sum of the indent number of line 3 plus 1, which is “2”.Further, if line 4 also has subordinate lines, the same process isperformed for these lines as well.

[0262] Third, in this process, with the moving of line 3 and line 4, theline starting position of line 2 also needs to be changed. The new linestarting position of line 2 is obtained by the formula y2=y4+h4+α,wherein y4 denotes the y coordinate value of the line starting positionof line 4, h4 denotes the height of the textbox of line 4, y2 denotesthe y coordinate value of the line starting position of line 2, and αdenotes the distance in between the lines, which is arbitrarilydetermined by the user. In the case where there are more lines underline 2, the above-described process is repeated for each of these linesas well. Since all the results from the above processes are stored inthe result memory 102, the process can then move on to the displayprocess wherein the results can be displayed on the display apparatus 2₂. As a result of the above-described process, the indent positions ofthe lines can be corrected as shown in FIG. 48B.

[0263] Next, the output process, which is the second one of the outputdestinations of the user operation determination process, is describedin further detail.

[0264]FIG. 49 is a flow chart for illustrating the output process.

[0265] The output process of step S90 realizes the storing of therecognized character strings and the outline information stored in theresult memory 102 into a storage medium such as a hard disk or anoptical disk, or the transmission of the above information data to adesignated person as an e-mail.

[0266] First, as shown in FIG. 49, the information stored in the resultmemory is converted into text through a format conversion process (stepS91). The data in the result memory 102 are called in order from theline data with the line starting position having the smallest ycoordinate value and converted into a file format such as html text orTAB text based on the indent number. The user then inputs the necessaryinformation for outputting. The above information would be a file namein the case of storing the data in a storage medium, and an e-mailaddress and a title in the case of sending the data through a network.The data converted into text in the manner described above are output toa storage medium 4 and a network 5. When the data are output, theprocess goes back to the recognition instruction discrimination processof step S10 so that handwriting input can be newly performed (step S92).Note that the application used for outputting the data to the network 5is e-mail software such as Outlook and the like.

[0267] According to the above embodiment of the present invention,handwritten information is stored as stroke information and lineinformation and can be batch converted at an arbitrary timing chosen bythe user, rather than progressively converting the handwrittencharacters into type. Also, the indent position can be obtained from theposition at which the handwriting information is input so that a formaloutline of the character strings can be created, and further, the formaloutline may be easily corrected by the user.

[0268] Additionally, according to the present embodiment, the aboveoutlined character strings can be stored in a storage medium or sent tosomeone via e-mail upon entering an e-mail address and a title.

[0269] Here, it is noted that the present invention, is not limited tothe above-described preferred embodiments, and various changes andmodifications are possible without departing from the scope of thepresent invention. For example, according to the above-describedembodiments, the handwriting input/display apparatus 2 and the computer1 are configured to be two separate components; however, the computermay be implemented within the handwriting input/display apparatus sothat the handwriting input/display apparatus itself becomes the actualhandwriting information processing apparatus.

[0270] Finally, this patent application claims the benefit of theearlier filing date of Japanese Patent Application No. 2002-002946 filedJan. 10, 2002, and is based on Japanese Patent Application No.2002-002946 and Japanese Patent Application No. 2001-292620, the entirecontents of which are hereby incorporated by reference.

What is claimed is:
 1. A handwriting information processing apparatuscomprising: a coordinate input part that detects a handwriting inputmade by a user and outputs a coordinate value; a processing part thatperforms predetermined processes based on said coordinate value outputfrom said coordinate input part; and a display part that displays anoutput of said processing part; wherein said processing part comprises:a coordinate value storing part that stores all of the coordinate valuesoutput by said coordinate input part along with stroke information andline information of said handwriting input; and a handwritten characterrecognition part that recognizes handwritten characters in a batch basedon the coordinate values, the stroke information, and the lineinformation stored in said coordinate value storing part.
 2. Thehandwriting information processing apparatus as claimed in claim 1,wherein said processing part further comprises a Chinese characterconversion part that performs a Chinese character conversion by handinga character string obtained by said handwritten character recognitionpart to an Input Method Editor.
 3. The handwriting informationprocessing apparatus as claimed in claim 1, wherein said processing partfurther comprises: an outline drafting part that calculates an indentposition of each line based on the coordinate values and the lineinformation stored in said coordinate value storing part, obtainsoutline information of a character string, and drafts an outline; aresult storing part that stores the character string obtained by saidhandwritten character recognition part and the outline informationobtained by said outline drafting part; and an edit processing part thatcorrects said outline information based on information stored in saidresult storing part.
 4. The handwriting information processing apparatusas claimed in claim 3, further comprising: an outline draft storing partthat stores said character string of the drafted outline in a storagemedium; and an outline draft transmitting part that converts saidcharacter string of the drafted outline for e-mail transmission, andsends the converted outline draft to a designated e-mail address.
 5. Astorage medium that stores a program readable by a central processingunit that performs a handwriting character batch conversion methodcomprising a coordinate input step of detecting a handwriting input madeby a user and outputting a coordinate value, a processing step ofperforming predetermined processes based on said coordinate value outputby said coordinate input step, and a display step of displaying anoutput from said processing step, wherein said processing stepcomprises: a coordinate value storing step of storing all of thecoordinate values output by, said coordinate input step along withstroke information and line information of said handwriting input; and ahandwritten character recognition step of recognizing handwrittencharacters in a batch based on the coordinate values, the strokeinformation, and the line information stored by said coordinate valuestoring step.
 6. The storage medium as claimed in claim 5, wherein saidprocessing step further comprises a Chinese character conversion step ofperforming a Chinese character conversion by handing a character stringobtained by said handwritten character recognition step to an InputMethod Editor.
 7. The storage medium as claimed in claim 5, wherein saidprocessing step further comprises: an outline drafting step ofcalculating an indent position of each line based on the coordinatevalues and the line information stored by said coordinate value storingstep, obtaining outline information of a character string, and draftingan outline; a result storing step of storing the character stringobtained in said handwritten character recognition step and the outlineinformation obtained in said outline drafting step; and an editprocessing step of correcting said outline information based oninformation stored by said result storing step.
 8. The storage medium asclaimed in claim 7, wherein said handwriting character batch conversionmethod further comprises: an outline draft storing step of storing saidcharacter string of the drafted outline in a storage medium; and anoutline draft transmitting step of converting said character string ofthe drafted outline for e-mail transmission, and sending the convertedoutline draft to a designated e-mail address.
 9. A program that performsa handwriting character batch conversion method comprising a coordinateinput step of detecting a handwriting input made by a user andoutputting a coordinate value, a processing step of performingpredetermined processes based on said coordinate value output by saidcoordinate input step, and a display step of displaying an output fromsaid processing step, wherein said processing step further comprises: acoordinate value storing step of storing all of the coordinate valuesoutput by said coordinate input step along with stroke information andline information of said handwriting input; and a handwritten characterrecognition step of recognizing handwritten characters in a batch basedon the coordinate values, the stroke information, and the lineinformation stored by said coordinate value storing step.
 10. Theprogram as claimed in claim 9, wherein said processing step furthercomprises a Chinese character conversion step of performing a Chinesecharacter conversion by handing a character string obtained by saidhandwritten character recognition step to an Input Method Editor. 11.The program as claimed in claim 9, wherein said processing step furthercomprises: an outline drafting step of calculating an indent position ofeach line based on the coordinate values and the line information storedby said coordinate value storing step, obtaining outline information ofa character string, and drafting an outline; a result storing step ofstoring the character string obtained in said handwritten characterrecognition step and the outline information obtained in said outlinedrafting step; and an edit processing step of correcting said outlineinformation based on information stored by said result storing step. 12.The program as claimed in claim 11, wherein said handwriting characterbatch conversion method further comprises: an outline draft storing stepof storing said character string of the drafted outline in a storagemedium; and an outline draft transmitting step of converting saidcharacter string of the drafted outline for e-mail transmission, andsending the converted outline draft to a designated e-mail address.